Testing parallel applications using code instrumentation

ABSTRACT

A method for testing a computer program that includes software code arranged to be executed in multiple threads, including at least a first thread and one or more other threads. The method includes identifying an output statement in the first thread, running the software code so as to generate an output due to the output statement, and intervening in the program so that when the program is run, execution of the first thread is blocked prior to the output statement while the one or more other threads continue to execute. The output generated after intervening in the program is examined in order to detect a fault in the program.

FIELD OF THE INVENTION

[0001] The present invention relates generally to software testing, andspecifically to testing of parallel software applications.

BACKGROUND OF THE INVENTION

[0002] Sequential software, which runs on a single platform in a singlethread, executes in a deterministic order. In other words, given thesame input, the sequence of statement execution is fixed and unvaryingin repeated executions of the software.

[0003] In parallel software applications, on the other hand, the orderof execution of program statements may vary from one run to the next. Inthe context of the present patent application and in the claims,parallel software includes any sort of multi-threaded, concurrent, ordistributed software, and may run on a single processor or on multipleprocessors. In parallel software, the sequence of statement execution isdependent, inter alia, on scheduler decisions, order of message arrival,synchronization mechanisms, and relative speed of hardware involved.Whereas in sequential software, the program output is uniquelydetermined by the inputs selected, in the case of parallel software, theoutputs may depend not only on the input space of the program, but alsoon the order in which different tasks are performed. The set ofinformation that describes a sequence in which a parallel programexecutes in a given execution run is called an interleaving. Faults in aparallel application, such as race conditions, may manifest themselvesin one interleaving but not in others, making the task of debugging theapplication all the more difficult.

[0004] In response to this difficulty, testing tools for parallelapplications have been developed that are based on adding noise to theapplication. The noise changes the timing of the application, in anattempt to expose timing bugs that arise when an implementation does notconsider a specific interleaving in which a fault is manifested. Forexample, Edelstein et al. describe a tool of this sort, known asConTest, for detecting synchronization faults, in an article entitledMultithreaded Java Program Test Generation, IBM Systems Journal 41:1(2002), pages 111-125, which is incorporated herein by reference. A Javaapplication program under test is seeded with a sleep( ), yield( ) orpriority( ) primitive at shared memory accesses and synchronizationevents. At run time, ConTest makes random or coverage-based decisions asto whether the seeded primitive is to be executed. The probability offinding concurrent faults is thus increased. A replay algorithmfacilitates debugging by saving the order of shared memory accesses andsynchronization events. A suitable replay algorithm for this purpose isdescribed by Choi et al., in Deterministic Replay of Java MultithreadedApplications, Proceedings of the SIGMETRICS Symposium on Parallel andDistributed Tools (ACM, New York 1998), pages 48-59, which isincorporated herein by reference.

SUMMARY OF THE INVENTION

[0005] The present invention provides improved tools for detectingfaults in parallel software programs, based on an adversarial approachthat attempts to induce failures at such points. Embodiments of thepresent invention are described herein with particular reference tomultithreaded Java applications, but the principles of the presentinvention may similarly be applied to concurrent, distributed andmultithreaded programs of other sorts, as well. The term thread, as usedin the present patent application and in the claims, should therefore beunderstood broadly to refer to any sequence of program statements thatis executed in order by a processor, in a computing environment in whichmultiple sequences (threads) of this sort are executed in parallel aspart of a single program. Thus, for example, in the present context,threads also include entities that are commonly referred to as processesand tasks. The software program under test may be designed for executionon a single processor or on a group of parallel processing nodes.

[0006] In embodiments of the present invention, a parallel applicationunder test is executed, and the locations of output statements in theapplication are logged. The application code prior to each of theseoutput statements is then instrumented in order to delay the timing ofthe output statements, and the application is executed again. In otherwords, if an output statement occurs in one thread of the application,instrumentation of the code causes that thread to be stopped, while theother threads are allowed to proceed. The other threads typicallyproceed as far as possible, until a point is reached at which thestopped thread is the only one that can run (and therefore must beallowed to continue). The execution of the instrumented code is logged,as well, and any differences in the outputs from those logged in theoriginal execution run are noted. Thus, if there are differentinterleavings of the application program that generate different outputseven interleavings with a very low probability of occurring in practicethey will be uncovered by the tools provided by the present invention.

[0007] Embodiments of the present invention can also be used in testingsoftware that may have multiple correct outputs. In this case, thedifferent outputs generated by execution of the instrumented code may bechecked against a set of valid outputs, rather than just against theoriginal output generated by the non-instrumented code. Alternatively oradditionally, instrumentation of the application as provided by thepresent invention may be used to facilitate testing of a particularexecution sequence, among multiple possible execution sequences of theapplication, by delaying other execution sequences so that the sequenceof interest can execute.

[0008] Although the embodiments described herein are based mainly ontechniques of software instrumentation, other types of user exitimplementation may likewise be used to intervene in the program in orderto block certain program threads. Such alternative thread-blockingtechniques are also considered to be within the scope of the presentinvention.

[0009] There is therefore provided, in accordance with an embodiment ofthe present invention, a method for testing a computer program thatincludes software code arranged to be executed in multiple threads,including at least a first thread and one or more other threads, themethod including:

[0010] identifying an output statement in the first thread;

[0011] running the software code so as to generate an output due to theoutput statement;

[0012] intervening in the program so that when the program is run,execution of the first thread is blocked prior to the output statementwhile the one or more other threads continue to execute; and

[0013] examining the output generated after intervening in the programin order to detect a fault in the program.

[0014] Identifying the output statement typically includes running theprogram without intervening in the program, and logging the output ofthe program in order to locate the output statement. Examining theoutput may include comparing the output generated after intervening inthe program to the output logged before intervening in the program, anddetecting a difference in the output.

[0015] Additionally or alternatively, examining the output includescomparing the output to a set of allowed outputs, and detecting thefault when the output is not a member of the set.

[0016] In embodiments of the invention, intervening in the programincludes instrumenting the software code. Typically, identifying theoutput statement includes identifying multiple output statements in oneor more of the threads including at least the first thread, andinstrumenting the software code includes inserting a respective codeinstruction prior to each of the identified output statements.

[0017] Additionally or alternatively, instrumenting the software codeincludes blocking the first thread so that the execution of the firstthread will resume only when none of the multiple threads, other thanthe first thread, is able to continue to execute. In disclosedembodiments, blocking the first thread includes inserting at least oneof a sleep( ) statement and a yield( ) statement in the first threadprior to the output statement.

[0018] In one embodiment, instrumenting the software code includesinserting a code instruction to block the first thread at a location inthe software code immediately prior to the output statement. In anotherembodiment, the output includes a value of a variable that is set at anassignment statement preceding the output statement in the first thread,and instrumenting the software code includes inserting a codeinstruction to block at least one of the first thread and the otherthreads at a location in the software code between the assignmentstatement and the output statement. In a further embodiment, identifyingthe output statement includes identifying a block in the first threadthat begins with an enter block statement and contains the outputstatement, and instrumenting the software code includes inserting a codeinstruction to block the first thread prior to the enter blockstatement.

[0019] The multiple threads may include at least one of a process and atask, and running the software code may include running the softwarecode on multiple processing nodes in parallel.

[0020] There is also provided, in accordance with an embodiment of thepresent invention, a method for testing a computer program that includessoftware code arranged to be executed in multiple threads and havingmultiple possible execution flows, which are characterized by respectivesequences of outputs, the method including:

[0021] selecting a desired execution flow, characterized by a desiredsequence of outputs, from among the multiple possible execution flows;

[0022] running the software code;

[0023] intervening in the program so that when the program is run,execution of the threads leading to the outputs that are not in thedesired sequence is blocked, while one or more of the threads continueto execute in the desired execution flow; and

[0024] testing the desired execution flow while running the softwarecode after intervening in the program.

[0025] In one embodiment, intervening in the program includesinstrumenting the software code by inserting program statements in thethreads leading to the outputs that are not in the desired sequence sothat the execution of the threads leading to the outputs that are not inthe desired sequence will resume only when none of the one or more ofthe threads is able to continue to execute in the desired executionflow.

[0026] There is additionally provided, in accordance with an embodimentof the present invention, apparatus for testing a computer program thatincludes software code arranged to be executed in multiple threads,including at least a first thread and a one or more other threads, theapparatus including at test workstation, which is arranged to identifyan output statement in the first thread, and to intervene in the programso that when the program is run, execution of the first thread isblocked prior to the output statement while the one or more otherthreads continue to execute, the workstation being further arranged torun the software code so as to generate an output due to the outputstatement, wherein the output is indicative of a fault in the program.33. There is further provided, in accordance with an embodiment of thepresent invention, apparatus for testing a computer program thatincludes software code arranged to be executed in multiple threads andhaving multiple possible execution flows, which are characterized byrespective sequences of outputs, the apparatus including a testworkstation, which is arranged to select a desired execution flow,characterized by a desired sequence of outputs, from among the multiplepossible execution flows, and to intervene in the program so that whenthe program is run, execution of the threads leading to the outputs thatare not in the desired sequence is blocked, while one or more of thethreads continue to execute in the desired execution flow, so that uponrunning the software code, the desired execution flow is tested.

[0027] There is moreover provided, in accordance with an embodiment ofthe present invention, a computer software product for testing anapplication program that includes software code arranged to be executedin multiple threads, including at least a first thread and one or moreother threads, the product including a computer-readable medium in whichprogram instructions are stored, which instructions, when read by acomputer, cause the computer to identify an output statement in thefirst thread, and to intervene in the program so that when theapplication program is run, execution of the first thread is blockedprior to the output statement while the one or more other threadscontinue to execute, the instructions further causing the computer torun the software code so as to generate an output due to the outputstatement, wherein the output is indicative of a fault in the program.

[0028] There is furthermore provided, in accordance with an embodimentof the present invention, a computer software product for testing acomputer program that includes software code arranged to be executed inmultiple threads and having multiple possible execution flows, which arecharacterized by respective sequences of outputs, the product includinga computer-readable medium in which program instructions are stored,which instructions, when read by a computer, cause the computer toselect a desired execution flow, characterized by a desired sequence ofoutputs, from among the multiple possible execution flows, and tointervene in the program so that when the program is run, execution ofthe threads leading to the outputs that are not in the desired sequenceis blocked, while one or more of the threads continue to execute in thedesired execution flow, so that upon running the software code, thedesired execution flow is tested.

[0029] The present invention will be more fully understood from thefollowing detailed description of the embodiments thereof, takentogether with the drawings in which:

BRIEF DESCRIPTION OF THE DRAWINGS

[0030]FIG. 1 is a schematic, pictorial illustration of a system forsoftware testing, in accordance with an embodiment of the presentinvention;

[0031]FIG. 2 is a flow chart that schematically illustrates a method fortesting a software application, in accordance with an embodiment of thepresent invention;

[0032]FIG. 3 is a graph that schematically illustrates an executionsequence of a multithreaded software application under test, inaccordance with an embodiment of the present invention; and

[0033]FIG. 4 is a flow chart that schematically illustrates a method fortesting a software application, in accordance with an alternativeembodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

[0034]FIG. 1 is a schematic, pictorial illustration of a system 20 forsoftware testing, in accordance with an embodiment of the presentinvention. System 20 is built around a test workstation 22, typically ageneral-purpose computer with suitable software for carrying out thefunctions described hereinbelow. This software may be downloaded toworkstation 22 in electronic form, over a network, for example, or itmay alternatively be furnished on tangible media, such as CD-ROM. Thesoftware test functions of the present invention may be combined insystem 20 with other testing tools, such as the above-mentioned ConTesttool and race detection functions described in U.S. Patent applicationSer. No. 10/035,422, filed Jan. 4, 2002, which is assigned to theassignee of the present patent application and whose disclosure isincorporated herein by reference.

[0035] System 20 is designed to receive and test a parallel softwareapplication 24. In the present example, application 24 is assumed tocomprise a multithreaded Java application, which may be provided insource code or byte code form. A test program 26, which is alsotypically written in Java as either a single-threaded or multithreadedprogram, is used to run application 24 with a given set of inputs andexecution conditions. The execution of the test is recorded in a log 28,including all outputs generated by the application. The logged resultsare used in instrumenting the code of application 24 for furthertesting, as described below. The log may also be used to hold and replaythe order of memory access and synchronization events, to aid indebugging application 24 when faults are found. This replay function maybe carried out using the algorithm described by Choi et al. in theirabove-mentioned article, or using other tools known in the art, such asthe Rational Robot, offered by Rational Software (Cupertino, Calif.).

[0036]FIG. 2 is a flow chart that schematically illustrates a method fortesting application 24, in accordance with an embodiment of the presentinvention. Initially, workstation 22 executes multithreaded application24 using test program 26, at an initial execution step 30. The resultsof the test are recorded in log 28, at a logging step 32, including thelocations of all output statements in application 24. Workstation 22then automatically instruments the code of application 24 (typically theJava byte code) prior to each output statement, at an instrumentationstep 34. The purpose of this instrumentation is to delay the threadcontaining the output statement for as long as possible, typically untilthe stopped thread is the only one that can still run. Various differentmethods may be used to induce the desired delay. Some exemplary methodsare described below.

[0037] After instrumenting the code, workstation 22 runs application 24again, using test program 26, at a repeat execution step 36. The outputsof the application are again recorded in log 28. The workstation checksthese outputs, at an output checking step 38. If application 24 issupposed to give a unique sequence of outputs (for the particular set ofinputs specified by test program 26), then it is sufficient at step 38to check the outputs of the instrumented application against the outputsoriginally logged at step 32. The application is considered to havepassed the test if the outputs are unchanged by the instrumentation.Otherwise, the application is debugged, typically by the programmer whocreated it, in order to remove the timing fault that caused theinconsistent outputs at a debugging step 40.

[0038] Alternatively, in some applications, the expected output sequencefor a given set of inputs may not be unique. In this case, workstation22 may need to resort to an oracle a program (possibly created by ahuman expert) capable of determining which output sequences are correct.Variations in the output sequences permitted by the oracle typicallyinclude certain reorderings of the outputs, without changes in theirvalues, but the variations may also include changes in the output valuesthemselves. In either case, the oracle is preferably capable ofverifying, for any sequence of outputs, whether or not the sequencebelongs to the set of legal outputs of the test applied by test program26.

[0039]FIG. 3 is a graph that schematically illustrates the order ofexecution of a multithreaded software application 50 under test insystem 20, in accordance with an embodiment of the present invention.The application comprises two threads 52 and 54, which are initiated bya main program 56 and comprise respective sequences of programstatements 58. At a first assignment statement 60 in thread 52, avariable Y is assigned the value Z. Under ordinary operating conditions,an output statement 62 (PRINT Y) in thread 52 will cause the applicationto output the value Z. Later on, a second assignment statement 64 inthread 54 will assign Y the value X. It can be seen in FIG. 3 that ifthread 52 is delayed between statements 60 and 62, statement 62 willcause the application to output the value X, but this fault may not havebeen detected by the application programmer.

[0040] Therefore, in order to reveal the faulty behavior of application50, workstation 22 instruments the application program at step 34 inorder to delay statement 62 for as long as possible. The codeinstrumentation can be expressed in pseudocode form as follows: Haltingthread:: otherAdvancing = true While(otherAdvancing) {otherAdvancing =false; sleep(duration) }; Other threads execute: otherAdvancing = true;

[0041] In a simple implementation shown in FIG. 3, a sleep( ) statement66 is added to the program just before step 62. Statement 66 causesthread 52 to be blocked until thread 54 (as well as any other threads inthe program) is blocked or finishes its execution. (This implementationof sleep( ) differs from that described in the above-mentioned articleby Edelstein et al., in which the sleep( ) statement causes a thread tobe blocked for a randomly selected period of time.) Alternatively, ayield( ) statement may be used at statement 66 (in place of sleep( )),in order to pass control from thread 52 to thread 54. Assuming there isno point in thread 54 prior to statement 64 at which thread 54 will beblocked while waiting for thread 52 to execute, the addition ofstatement 66 to thread 52 will cause the unexpected value X to beprinted at statement 62.

[0042] Alternatively, it may be desirable to use a more compleximplementation at step 34, since by the time the program flow reachesoutput statement 62 in step 36, it may be too late to actually changethe output order. Thus, statement 66 may be introduced in thread 52 atsome other stage, for example, directly following the last point inthread 52 before output statement 62 at which variable Y is set (i.e.,just after assignment statement 60), or intermediate between theassignment and output statements. Program analysis techniques known inthe art, such as program slicing, may be used to detect the instructionsthat precede the output statement. Program slicing is described, forexample, by Lev-Ami et al., in Putting Static Analysis to Work forVerification: A Case Study, in Proceedings of the ACM InternationalSymposium on Software Testing and Analysis (ISSTA) (2000), pages 26-38,which is incorporated herein by reference.

[0043] Although in the simplified example shown in FIG. 3, assignmentstatement 60 is in the same thread 52 as output statement 62, in othercases the assignment and output statements may belong to differentthreads. In such cases, the thread containing the assignment may beblocked before the assignment is executed (so that it may remain blockeduntil after the output occurs), or the thread containing the outputstatement may be blocked at the statement that it happens to havereached the assignment executes. Even more generally, it is possible toidentify a set of statements on which the output depends, directly orindirectly, in whatever thread these statements occur, and block thisthread before the assignment, or block one of its dependent threads.

[0044] As another example, two different threads that are synchronizedon the same object will be mutually exclusive, so that whichever threadis first to acquire a lock on the object will execute first. In order tochange the order of execution, workstation 22 may search at step 34 foran enter block statement, such as the Java byte code statementmonitorenter(obj), in the thread that executed first at step 30 (whereinobj is the synchronization object of the two threads). A sleep( ) orother delay statement may then be instrumented into the code before theenter block statement in order to force a change in the thread executionorder.

[0045] Other methods for delaying a thread prior to an output statementwill be apparent to those skilled in the art and are considered to bewithin the scope of the present invention.

[0046]FIG. 4 is a flow chart that schematically illustrates a method fortesting application 24, in accordance with an alternative embodiment ofthe present invention. This embodiment uses the principles of thepresent invention in a different way not to induce failure in testing aparallel application, but rather to cause a test of the application tofollow a certain desired execution path (which may result either insuccessful execution or failure). This feature of the present inventionis useful, for example, in conjunction with test generation tools, suchas the GOTCHA-TCBeans Software Test Tool Kit, developed by InternationalBusiness Machines Corporation (Armonk, N.Y.). Tools of this sort attemptto predict the sequence of states or events that a program under testwill traverse, and then automatically create tests to evaluate theprogram, assuming the program follows this predicted sequence. Whenmultiple legal execution sequences are possible, however, the actualexecution of the program may not follow the predicted sequence, and thetests therefore will not give useful results.

[0047] To prevent this eventuality, workstation 22 analyzes test program26 to determine the desired execution flow of application 24 under test,at a flow analysis step 70. The test program may be supplied by a humanuser, such as a test engineer, or alternatively, it may be created by anautomated test generation tool, as mentioned above. For example, assumethat the sequence of outputs that may be generated by application 24 isA→B→ (C,D,E) and C→F. In other words, output A is followed by output B,which may legally be followed by any of outputs C, D and E. If output Coccurs, it is expected to be followed by output F. The desired executionflow under test program 26 is assumed to be the sequence A→B→C→F.Although outputs D and E are legal, they are not desirable in theframework of test program 26.

[0048] To facilitate successful execution of the test, workstation 22locates branch points in application 24 statements in the threads of theapplication that lead to outputs D and E, at a branch location step 72.The workstation then instruments application 24, in a manner similar tothat described above, in order to delay the threads that lead to theseundesired outputs, at a thread delay step 74. The workstation runs theinstrumented application, using test program 26, at a test executionstep 76. As a result of the instrumentation, the desired sequenceA→B→C→F is observed and can be evaluated for success or failure.

[0049] Although the embodiments described hereinabove are based mainlyon techniques of software instrumentation, other types of user exitimplementation may likewise be used to intervene in the program in orderto block selected program threads. For example, when an applicationprogram under test is executed using another program (such as a JavaVirtual Machine—JVM), this other program may be modified to detect whena certain thread in the application program is running and to allowthreads to be blocked. A technique known as wrapping may be used in asimilar fashion, when the application program under test makes use of asynchronize system method. In this case, instead of instrumenting theapplication program code, a new synchronize method can be provided,which allows intervention in the program in addition to executing thesystem synchronize. (This technique of intervention is also known asoverloading, although it is not known in the art to use wrapping oroverloading for the purposes of the present invention.) Other methodsfor intervening in the control flow of an application, which can be usedto block a selected thread, will be apparent to those skilled in the artand are considered to be within the scope of the present invention.

[0050] It will be appreciated that the embodiments described above arecited by way of example, and that the present invention is not limitedto what has been particularly shown and described hereinabove. Rather,the scope of the present invention includes both combinations andsubcombinations of the various features described hereinabove, as wellas variations and modifications thereof which would occur to personsskilled in the art upon reading the foregoing description and which arenot disclosed in the prior art.

1. A method for testing a computer program that includes software codearranged to be executed in multiple threads, including at least a firstthread and one or more other threads, the method comprising: identifyingan output statement in the first thread; running the software code so asto generate an output due to the output statement; intervening in theprogram so that when the program is run, execution of the first threadis blocked prior to the output statement while the one or more otherthreads continue to execute; and examining the output generated afterintervening in the program in order to detect a fault in the program. 2.The method according to claim 1, wherein identifying the outputstatement comprises running the program without intervening in theprogram, and logging the output of the program in order to locate theoutput statement.
 3. The method according to claim 2, wherein examiningthe output comprises comparing the output generated after intervening inthe program to the output logged before intervening in the program, anddetecting a difference in the output.
 4. The method according to claim1, wherein examining the output comprises comparing the output to a setof allowed outputs, and detecting the fault when the output is not amember of the set.
 5. The method according to claim 1, whereinintervening in the program comprises instrumenting the software code. 6.The method according to claim 5, wherein identifying the outputstatement comprises identifying multiple output statements in one ormore of the threads comprising at least the first thread, and whereininstrumenting the software code comprises inserting a respective codeinstruction prior to each of the identified output statements.
 7. Themethod according to claim 5, wherein instrumenting the software codecomprises blocking the first thread so that the execution of the firstthread will resume only when none of the multiple threads, other thanthe first thread, is able to continue to execute.
 8. The methodaccording to claim 7, wherein blocking the first thread comprisesinserting at least one of a sleep( ) statement and a yield( ) statementin the first thread prior to the output statement.
 9. The methodaccording to claim 5, wherein instrumenting the software code comprisesinserting a code instruction to block the first thread at a location inthe software code immediately prior to the output statement.
 10. Themethod according to claim 5, wherein the output comprises a value of avariable that is set at an assignment statement preceding the outputstatement in the first thread, and wherein instrumenting the softwarecode comprises inserting a code instruction to block at least one of thefirst thread and the other threads at a location in the software codebetween the assignment statement and the output statement.
 11. Themethod according to claim 5, wherein identifying the output statementcomprises identifying a block in the first thread that begins with anenter block statement and contains the output statement, and whereininstrumenting the software code comprises inserting a code instructionto block the first thread prior to the enter block statement.
 12. Themethod according to claim 1, wherein the multiple threads comprise atleast one of a process and a task.
 13. The method according to claim 1,wherein running the software code comprises running the software code onmultiple processing nodes in parallel.
 14. A method for testing acomputer program that includes software code arranged to be executed inmultiple threads and having multiple possible execution flows, which arecharacterized by respective sequences of outputs, the method comprising:selecting a desired execution flow, characterized by a desired sequenceof outputs, from among the multiple possible execution flows; runningthe software code; intervening in the program so that when the programis run, execution of the threads leading to the outputs that are not inthe desired sequence is blocked, while one or more of the threadscontinue to execute in the desired execution flow; and testing thedesired execution flow while running the software code after interveningin the program.
 15. The method according to claim 14, whereinintervening in the program comprises instrumenting the software code.16. The method according to claim 15, wherein instrumenting the softwarecode comprises running the program before instrumenting the softwarecode, and logging the outputs of the program in order to determine oneor more locations at which the code should be instrumented.
 17. Themethod according to claim 15, wherein instrumenting the software codecomprises inserting program statements in the threads leading to theoutputs that are not in the desired sequence so that the execution ofthe threads leading to the outputs that are not in the desired sequencewill resume only when none of the one or more of the threads is able tocontinue to execute in the desired execution flow.
 18. The methodaccording to claim 14, wherein the multiple threads comprise at leastone of a process and a task.
 19. The method according to claim 14,wherein running the software code comprises running the software code onmultiple processing nodes in parallel.
 20. Apparatus for testing acomputer program that includes software code arranged to be executed inmultiple threads, including at least a first thread and a one or moreother threads, the apparatus comprising at test workstation, which isarranged to identify an output statement in the first thread, and tointervene in the program so that when the program is run, execution ofthe first thread is blocked prior to the output statement while the oneor more other threads continue to execute, the workstation being furtherarranged to run the software code so as to generate an output due to theoutput statement, wherein the output is indicative of a fault in theprogram.
 21. The apparatus according to claim 20, wherein theworkstation is arranged to identify the output statement by running theprogram without intervening in the program, and logging the output ofthe program in order to locate the output statement.
 22. The apparatusaccording to claim 1, wherein the workstation is arranged to find thefault in the program by comparing the output generated after interveningin the program to the output logged before intervening in the program,and detecting a difference in the output.
 23. The apparatus according toclaim 20, wherein the workstation is arranged to find the fault in theprogram by comparing the output to a set of allowed outputs, anddetecting the fault when the output is not a member of the set.
 24. Theapparatus according to claim 20, wherein the workstation is adapted tointervene in the program by instrumenting the software code.
 25. Theapparatus according to claim 24, wherein the workstation is arranged toidentify multiple output statements in one or more of the threadscomprising at least the first thread, and to instrument the code byinserting a respective code instruction in the code prior to each of theidentified output statements.
 26. The apparatus according to claim 24,wherein the workstation is arranged to instrument the software code inorder to block the first thread so that the execution of the firstthread will resume only when none of the multiple threads, other thanthe first thread, is able to continue to execute.
 27. The apparatusaccording to claim 26, wherein the workstation is arranged to instrumentthe software code by inserting at least one of a sleep( ) statement anda yield( ) statement in the first thread prior to the output statement.28. The apparatus according to claim 24, wherein the workstation isarranged to instrument the software code by inserting a code instructionto block the first thread at a location in the software code immediatelyprior to the output statement.
 29. The apparatus according to claim 24,wherein the output comprises a value of a variable that is set at anassignment statement preceding the output statement in the first thread,and wherein the workstation is arranged to instrument the software codeby inserting a code instruction to block at least one of the firstthread and the other threads at a location in the software code betweenthe assignment statement and the output statement.
 30. The apparatusaccording to claim 24, wherein the workstation is arranged to identify ablock in the first thread that begins with an enter block statement andcontains the output statement, and to instrument the software code byinserting a code instruction to block the first thread prior to theenter block statement.
 31. The apparatus according to claim 20, whereinthe multiple threads comprise at least one of a process and a task. 32.The apparatus according to claim 20, wherein the software code isarranged to run on multiple processing nodes in parallel.
 33. Apparatusfor testing a computer program that includes software code arranged tobe executed in multiple threads and having multiple possible executionflows, which are characterized by respective sequences of outputs, theapparatus comprising a test workstation, which is arranged to select adesired execution flow, characterized by a desired sequence of outputs,from among the multiple possible execution flows, and to intervene inthe program so that when the program is run, execution of the threadsleading to the outputs that are not in the desired sequence is blocked,while one or more of the threads continue to execute in the desiredexecution flow, so that upon running the software code, the desiredexecution flow is tested.
 34. The apparatus according to claim 33,wherein the workstation is adapted to intervene in the program byinstrumenting the software code.
 35. The apparatus according to claim34, wherein the workstation is arranged to run the program beforeinstrumenting the software code, and to log the outputs of the programin order to determine one or more locations at which the code should beinstrumented.
 36. The apparatus according to claim 34, wherein theworkstation is arranged to instrument the software code by insertingprogram statements in the threads leading to the outputs that are not inthe desired sequence so that the execution of the threads leading to theoutputs that are not in the desired sequence will resume only when noneof the one or more of the threads is able to continue to execute in thedesired execution flow.
 37. The apparatus according to claim 33, whereinthe multiple threads comprise at least one of a process and a task. 38.The apparatus according to claim 33, wherein the software code isarranged to run on multiple processing nodes in parallel.
 39. A computersoftware product for testing an application program that includessoftware code arranged to be executed in multiple threads, including atleast a first thread and one or more other threads, the productcomprising a computer-readable medium in which program instructions arestored, which instructions, when read by a computer, cause the computerto identify an output statement in the first thread, and to intervene inthe program so that when the application program is run, execution ofthe first thread is blocked prior to the output statement while the oneor more other threads continue to execute, the instructions furthercausing the computer to run the software code so as to generate anoutput due to the output statement, wherein the output is indicative ofa fault in the program.
 40. The product according to claim 39, whereinthe instructions cause the computer to identify the output statement byrunning the program without intervening in the program, and logging theoutput of the program in order to locate the output statement.
 41. Theproduct according to claim 40, wherein the instructions cause thecomputer to find the fault in the program by comparing the outputgenerated after intervening in the program to the output logged beforeintervening in the program, and detecting a difference in the output.42. The product according to claim 39, wherein the instructions causethe computer to find the fault in the program by comparing the output toa set of allowed outputs, and detecting the fault when the output is nota member of the set.
 43. The product according to claim 39, wherein theinstructions cause the computer to intervene in the program byinstrumenting the software code.
 44. The product according to claim 43,wherein the instructions cause the computer to identify multiple outputstatements in one or more of the threads comprising at least the firstthread, and to instrument the code by inserting a respective codeinstruction in the code prior to each of the identified outputstatements.
 45. The product according to claim 43, wherein theinstructions cause the computer to instrument the software code in orderto block the first thread so that the execution of the first thread willresume only when none of the multiple threads, other than the firstthread, is able to continue to execute.
 46. The product according toclaim 45, wherein the instructions cause the computer to instrument thesoftware code by inserting at least one of a sleep( ) statement and ayield( ) statement in the first thread prior to the output statement.47. The product according to claim 43, wherein the instructions causethe computer to instrument the software code by inserting a codeinstruction to block the first thread at a location in the software codeimmediately prior to the output statement.
 48. The product according toclaim 43, wherein the output comprises a value of a variable that is setat an assignment statement preceding the output statement in the firstthread, and wherein the instructions cause the computer to instrumentthe software code by inserting a code instruction to block at least oneof the first thread and the other threads at a location in the softwarecode between the assignment statement and the output statement.
 49. Theproduct according to claim 43, wherein the instructions cause thecomputer to identify a block in the first thread that begins with anenter block statement and contains the output statement, and toinstrument the software code by inserting a code instruction to blockthe first thread prior to the enter block statement.
 50. The productaccording to claim 39, wherein the multiple threads comprise at leastone of a process and a task.
 51. The product according to claim 39,wherein the software code is arranged to run on multiple processingnodes in parallel.
 52. A computer software product for testing acomputer program that includes software code arranged to be executed inmultiple threads and having multiple possible execution flows, which arecharacterized by respective sequences of outputs, the product comprisinga computer-readable medium in which program instructions are stored,which instructions, when read by a computer, cause the computer toselect a desired execution flow, characterized by a desired sequence ofoutputs, from among the multiple possible execution flows, and tointervene in the program so that when the program is run, execution ofthe threads leading to the outputs that are not in the desired sequenceis blocked, while one or more of the threads continue to execute in thedesired execution flow, so that upon running the software code, thedesired execution flow is tested.
 53. The apparatus according to claim52, wherein the instructions cause the computer to intervene in theprogram by instrumenting the software code.
 54. The product according toclaim 53, wherein the instructions cause the computer to run the programbefore instrumenting the software code, and to log the outputs of theprogram in order to determine one or more locations at which the codeshould be instrumented.
 55. The product according to claim 53, whereinthe instructions cause the computer to instrument the software code byinserting program statements in the threads leading to the outputs thatare not in the desired sequence so that the execution of the threadsleading to the outputs that are not in the desired sequence will resumeonly when none of the one or more of the threads is able to continue toexecute in the desired execution flow.
 56. The product according toclaim 52, wherein the multiple threads comprise at least one of aprocess and a task.
 57. The product according to claim 52, wherein thesoftware code is arranged to run on multiple processing nodes inparallel.